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ABSTBICT 

One of a series of sixteen* knowledge transf ormaticn 
papers r this paper deals with the nature and extent of use of 
e^alaati^ii data by administrators in vocational education. First, the 
follow ing eon d it ions which govern the uaa of the data are describedi . 
avallabiiity, reliability, credibility, utility, and consistency. 
Sirce the legitimacy of aost actions supported fcy evaluative data is 
obvious, it is not discussed, but one less wall recognized use, 
public relations, is acknowledged* llso explained ie the rolei ef 
evaluative cata in. state planning. While it is difficult to dfeteraini^ 
the actual e^i tent to which vocation al educators use evaluatiye data, ^ 
it is stated that sources do exi^t for this type of inforfflation, such 
as systeiatic analyses of state plans and surveys of state planners 
and adBinistrators. other evidence of the data's use is foun/d in the 
possibility to derive by Inference. Research is reported as /currently, 
underway en both the use and effectiveness of evaluative data* The 
state and local aanageient inforia/tion systems are desqribea since 
they are the iost extensively us e^^ data bases, ether potential 
gQurdeg are mentioned on the state and national level, particularly 
the new ' national Vocational iducation Data System and th^ N|ational 
and State Occupational inforfflation coordinating Cqmmittees* (ELG) 
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- - ■ 00spitr: the emphasis on ^^^ut^on Js^fi^ vocational educatlpn* 

: Uttl^ t$ knotm hpTSffyocati^nA^ the 

/ f|tf^t/ to whi(^ they use 
' ol thase d«s. Th^ to deat with tha/ lack of , , 

conunentaTy on the natUM and ,exte ©valuatlvii/^data by , /administra- 

tors ^of ydaatlon^i^ education* Topicj diatussed inciuda conditions govtming 
tlio Use of evaluative data, the Use of thWe.^ta for stite planning arid ^ 
iBipTOVeipent-^ofi^^ bf use of ttaWa data* and some ^ 

obsorvations^^T the ef foctivfness of and problems/in the use of evaluative 
Uttta in vaimronal tducatiopV' >^ l/ ^- * 

"UsiD Qf^^^aluative Data by Voeationai^ Edueators" ^/s ona of a slpries^ of 1$ 
pape^r'^^duced ^uring th^ /irst yfear of tha National Center's knowledge 
transToiTnatipn program; The 16 1 papers are eoncentrat^ in the fdur\hema 
areas emphasised under the NatloAal Center contraet^/^pec^ needs 
p^Ulatidns^ faimessV plahnlrig, and evaluation iii^ vocational education* ' 
The review and 'synthesis of research in each topic arfea is, intanded to com- 
municate know ledge iapd suggest Papers should be of interest 
to al 1 yocat ionar adUcators^ including administrators, re^eprcherSj federal 
agency personndiy and i:he staff. 



The pfcofesslon Is indebted tp Dr, Arthiir Lee for hi^ scholarship in 
prepinring th<e paper* Recognition iSy4lso due Dr, George M. Copa, University 
of Mrariepota, Dr,^ H, C, Ka^anas, Uni^er^y of Illinois, Champaign/Ur^na, 
and Dr, Norval McCaslin*, the ,Nation|il Center for Research, in Vocational 
liducation, for their critical review of the manuscript , Dr, Carol P, Kowle, 
research- specialist ,^ supervised the publication of the series, Ms, Jo-Ann 
Cherry coordinated editing and production, ^ 

■ •■ ■ [- ^ ■ . ■ / \ ' : ' ^ ^ 1 . , 

i\ , ' I. . ^ ^ Robert E, Taylor, ^ 

' , jf Exefcutive director 

; ' ■ _ - ^^tional C&^tet for Research 

in Vocational Education 



EKLC 



;' , ■ CONTENTS / • 

INTfidDUCTlON ,: ■ 

■ ^- ■ ■■ .--'.^ ■ ■ ; ■ ■ : 

CONDITIONS GOVERNING THE' USE OF EVALUATI^'DATA 
"^Availability ■ ' ' / ' 

Cre&lftTlty. ' 

■ Utnity 
Consistency . 

USES OF EVALUATIVE. DATA BY VOCATIONAL EDUCATORS 
Public/Relations , ■ 

State Planning 

EXTENT TO WHICH VOCATIONAL "EDUCATORS USE 
EVALUATIVE DATA '"**^^^ 

Sources of Information 

Otnferences < • , 

» 

EFFECTIVENESS IN THE USE OF EVALUATIVE DATA 

■ . I 

OBSERVATIONS ABOUT CURRENT EVALUATIVE DATA . 

;■ • ' / 

SUMMARY , ' 

Recommeridatlons 




INTRODUCTION 



•ffoBpito the emphasis on e\^luat ion and accounta^ilijty in ,vdcatid[nal education, 
litt lo Is known abput How. voestional educators yse evalirative d^a, the'exteht 
to which they use. the datar and the /esulta they obtain .from frtfe use of these 
data. The term evaluative date- refers to the pjogram'resultiJM^ed to measure 
direction and p^jress toward intenTOd objectives. ' Considerable rese^arch hps 
been conducted on approaches to evaluation, data to be obtained, .form^ and . 
instruments to tfe used, and* proceduTes to be folldwed* At the same time-,^ 
vary little has been published 05 the'urfe^f such research by ^vocatg^dnal* ' 
edufcatorSt ^ • 

There may be an assumption that research of this, kin^'is^ not needed, Edsall^ 
(V973),^in A GuidiP.for Loca^ Program Evaluation , says' that tKe use sdf evalua= 
t lon rosul& ^^sHbpd be^ obvibus''^(p. 11)'. Lamb C^P^) dlscasses a modfrl for 
e\?aluatiny yocational education \programs on the^bftsis of job placement and 
labor imirket supply and demand; She Htates, boweyerj that locating availabLc 
^rnto^Nmit Itin.^obtaining additional information, and analyzing Information ^r^ 
priinary^concernH; Forming judgments and making decisions are omitted in^the 
d ificussion^^^his .pattern is rep^ated^-^hroughQUt the literature on eva^^tio 
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CONDITWNS GOVERNING THE USE 
OF EVALUATIVE DATA 



r.. 



The present study is based on three assumptions: 



(L) the use of evaluative 



data cannot be assumed; (2) ''use"vis an active^ not passive functUofi; and 



(3) a numberof interrelated elements govern the uae of evaluative da^a'^^ 
Each assumpj^^' has an important role in the evaluatMn process. 

rhtv f ir.st,/fcslumpt lon should b^ se A-evident . The problemMnay be in the se 
aKHumptian* Whfen the term ''use'' ^&btaken to mean the function ^of judgment^ 
oniyj for €|xample, finding out which pvogi-ams are performing w$H and wj\^ch* 
ones are not^^. it 43 'easy assume that- evaluative data will^be used. Vlhen 
evaluative data lead to an' understanding of what certain^^Dgrams are acc^- 
plishdng and allpw conclusions to be drawn ar\d jAidgnients to b^ formedy the- 
data are obviously beneficiml'. But as^ long as their.~use/is ^^ssive ^ nothing^ 
Iu4s really ,happened, V^hen cohclusioftf^and judgments'VXead to^ action o£^ some 
kind/ If only to be incdrp6i*ated Into^he plajining process, there' is eyidenca, 
that w^aluative data hay 0 been put to use. \ * . 



*'UHe" in this h^se^iis nut a simple response to the^acqyisitionv^f evaluative 
data. ^ tew vocational educators command total conti 

legi^lat^ive actlfon,'^„^en' 
.yiy exercised withoift « 



implomenttft i'on ^fHheir programs* Administrative a 
ttie prerogatives of a. ^teacher in the dlassroom^ are 
mp advice or approva^ o#^t hers, , * 
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the US f Of ©valufltive data under th\jc circumstances is made increasingly . 
difficult asv^s^fi data encounter the^^c"r«tiny , and sometimes ^h#. hostility, 
of a varmy of interests. As Datta- (1978) says^*^*^tre is ;little fVidence.J 
that progrijns ,ara dismar^tled because of unfavorable evaluaticrts ; ^ 
avdluators have lamented that demons t^pbly 'lnfef fectiye progM^ on v 

political fieet undeterred by evidence'*' (p, 33) . , * . ' " 



As.Datta alsd notes,* however, there^ is "cause of hope** 'ip* 37y , Moreover^ 
there/^s evidence (bw, Vl976^ thrft e^s^aluative ^data are ^eing used, by / ' 
vocational educators e^n though their use is not belngAocumented, Yet^ 
^vocational . tducators sflfculd know ^why and how to TealiEe greater benefita from,, 
evaliiati^oo t J - ' : - . / 



I. 

Availabn ity 



Therd-are a fjunfoer of deterrer 
^4vairdbility% "^Researchers oi 



One is 
and have 

sys- 



si. 



Its to the use of , eValuative data. 
:eji develop good evaluation s^tems, 
' knowledge of the variety of p^ons who could use" the data, because th% 
^'tems 'are implemented by administrators responsible Jor the prMram\. 

This does not mean that every evaluation report should be publicised or dis- 
tributfed indiscriminately. Stlvenson anH W&rd {1975) are right in caiJtioning 



that , 



while 

< 



: jor 



4 ^ 



there are many uses, for ^valuation'pesMlts; . , tthis jfrs^ not to , ... 

* , * j£^ay*that , all; results, should immedi^ely be madBmm^d, Some 

^^^^^iesults are in-house items and |hould be used to/correct situa- ^. ^ 

, ^ ^'\tions prior to public disc^osura. (j^, 3) . . . ^ 

* ^' " f . ^ ' ^ ' ' ^'i ^ ^ I. " '5 ^ : ' ■ ^ • ■ ' ■ ^. 

.^Uslng evaluative data improperly in the -wrong ^context ,or without adequate ^ 
understanding of their implicatiohs is iworse than riot using them at 'all* 

' ■ VI. ^ : ^ ^ . ' - ^ ■ ' : . 

^ Thc^ayailabi 1 Lty of eMluptive data may only roquireiP that t^ey be presented 
in a way ^hat they can V5c readily understopd by ^diverjpe groups. . KJfefer and 
Voelknor' (197S)_T recogn i zed this when they developed an evaluation of voca-^ 
Ational triuc^tion special needs programs in Michigan. 'They commenced that 
constraints ^vflthih the State Department of Education necessitated keeping 
analyses simp^.W so that they coufd readily he interpreted' and used by persons 
^not th'dfrpughly fami^i^r wi^^^ ana^lysfs trchniquea, pWt% (1978) 

. [ also supports this observaW^ff when she cbmn\^ntSp "Evaluation r^Vts which 
' - ^re long on me^hodoiogy ^nd short' 'ori unqualified conclusions ar.e faiftires for 
most polldy purposes** [p, 14)./ The policy implications for the use of 
, evaluafive data in voQrftional educatiD|r ..are so sorious* tha^ this^=£^t.may be , 



respoXsible for the lack of detailed J Iterature on the subject* 
data can easily be misLteod. 



Evaluative 



I* 
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A ^ecor|d condition that govjerns" the use of eyalu^tiye data is their tejiibil-' 
it^* Reiearchers' usually ^meet this *requiroment , , *idwev|rs not' all '^valMativa 
data are^ direct or indirect rasults of reg^arch. There is no reason tcX 
assumt tHst .reljabia a^luatiye ^ta can* only obtained through^ researth.* 
in SQinei respects, the reieWch^ community rday be so inhibitad to' its concern * 
ovor data perfection that it reduc^'the contpibpti'oh it miglyt otherwise m^ke 
to the development of reasonably re^^^p Hata Venn (1978) commented at. a 
recent national conference/on measuring the^^outc^es^f vocational education! 

It'*ma>fbe thpt-=tbS search for evaluative da^^ ttmt are nonchante- ^ ' 
,able may ^become the so'arch that is unending and cpuld dead to the 
cul-do-lftc that methodology has often entered; if we can't measure 
it , isn't significant* '(p. 



'1 



I 



At %hQ same conferehce, Kilvit^ il91l0J\ pentioned a lack of datg'to provide 
'Mughlyv^re liable and valid answers,** but suggested f "it is , V^ifeverthe^lfesSi J* 
N valTiable to* ask what We do Jcnow and what c^ we re^sonabiy infej^ what 
we kilow** (p. 1).* . . ' ^ ^ ^ -''^ W 

Reliable evaluajDive data in this sen ^e are ^ot necessarily unimpea^^blq, 
^ There cahnot be obvious or .evW susp^ted. distortions , biasj oy carelessne|s 
A good example is the use of labor market, demand data* ASmi^iyfetrators hav^ 
regardej most state and local evaluations based on these datajtp^be without 
/merit. Their distrust of the data ^as confirmed by Wirt^ andTfeldfte^n 
(1975) , who stated: s i* ^ ^ ' ' i 



4n general ithe estimates of both omjUayment and unemployment 
currently being relied on t^ initiate irnese various programs Qn 
a state and smal le#*basis and to distribute multi-^Sillion dpl^r" 
federal appropriations are simply not reliable. (p.. 25/) 



HInco Wirtz and GO'ldstein have published their ^candid obsetva'ti9hs j^out the 1 
daui for which they once had official' respohsibilityv' the Depart m^t^ of Labor 
huH ' improved it^ dntu-goilerating. techniques, and -employment ^market 'cjata now^ / 
nvallabre t/ many states are'reportedly much more roliabre. ,Howq\4erj voca- 
t ionai^ etlut^itors are not likely to accept jthe ^lew data with- any more 
confidence Hhan they d^fl'^yho old* ^ y ' . ' *^*. * : . . ^ , 

- ^ - V. 4 ^ . : , X V/ ' ^ ^ ^ ^ ^ 

Credibility , - • ^ . ^ ^ 

CretybiLity i*s the third condition affecting the^ use of byaluatiV^ dati, 
DrevQS ^1978) illustrates its importance by dbscr^ag that **Local schools 
'With establish'^ .v^ational programs linked into Ae community infra-structure 
are npt going to be dissuaded from offering a progr^ sin)ply on the basis of 
output data sRbwi«g low ^racement rates ''in'occupp.tioW^^iged by an external^ 
standard to be relatfbd. to jfche program" (p. 10). Drewes and Katz (1975) have 
also noted U iscornableyt^ndency of vocntional^jptjiication^ professionals to 
ulistrust manpower data*' (p. 2,5)^ ^ ^ . "I \ ^ ' 1i / 
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TTietemre sevtral e^mplfiB of lack of credibility in vocational tdifeation 
evaluative data on a national scale, A task force appointed by HEW leill:*etary 
Eljiptt Richardsgn in 1971 fxamihe^ employment' eonditloni and the preparation 
^f yputh and adults for employment. The report ybrk in America^ (1972), was 
ba^ed on, the contributions of fifty-seven authors jsf cpnanissloinid papex^ and 
consultants and was circulated widely, 8he of its cohclusipns was that 
■'Vocational education ih the high schools has'* faijed tp give students useful ' 
skills or'pjace them in satisfying jobs" (p. 134), A broad generallMtion of 
this sort is indefensible and subject to loss of credibility^ ^ Rven Secretary 
Richardson could not give it his complete endorsement, and vocatioWl oducators 
were^ outspoken tn their criticism, . ^ 

'Later, the Associate Comrhissioner for Adult, Vocational, Technical and Manpower 
Education In th^ U,S, Office of Education appointed his own conunittee to cri- ^ 
tique the offensive chapter. The committee found fifteenVunacceptable 
^ assimptions In the report and proceededvito make a 'paragraph-by-paragraph 
•analysis of th,e content dealing with vocational education* * IWien they had 
finished, there was little vocational^ educators "qoul^ use in a study that had 
taken a year ^o complete, had cost a substantial amount of *money/and had been 
intended to hkve a major impact on federal policy, ^ * 

Another study two yetfrs later (WilmSj 197'4), sponsored by the Nation&l Inst^ 
tute^of Education (NIB) , reported two major findings, TTie first involved th^ 
, fact *that eight out of tan graduates of profesiional and lechnical-level post- 
secondary vocational programs did not get the jobs for which they trained. . 
T^ie second stated that eight ^ut of ten gMduates from lower-level progr.ams 
got t^ jobs they trained for but, with ^h# exception of secretaries, barely, 
earned the federal minimum wage* , ' ^ 

In response to thesj findings, critics noted that Wilms had been careless, in 
the way he hid gener$ldzed from dat« inadequate foT^hat purpose (Pucel, 1976;^ 
Magisos, 1976). Wilms^ and Hansell (1976) responded to these criticisms, but 
doing 10, appeared to contradict themselves,^ Wilms 'had reported his' find- 
ings se^parktely and apart from the qualifying clause',. **n6t representative of 
the entire; nation/'* Failure to admit that the findings were not representative 
destroyed the CMdlbllity of Wilms* report for vocational educators. The 
p^o^blem was not, ^s he states, the fact that "Wien stQdies uncover ^^vidence ^ 
that contradicts the conventional wisdom, vested Interests of ten challenge the 
Htiidv Josign and mothodology rather'than provide data to the contrary*' f( p. 22)* 

A more o'ecent ^ex^ple of lack of qredibility in evaluative, data appeared in the 
Phi Delta Kgippan . Egginton (1978) , reporting on a ^study of attitudes among 
' vocatianal education students, concluded; "It appears that participation in 
^vocational education programs is ineffective in developing a positive self-imag 
among the most alienated of the subgroups of teenagers** (p. 533), Swanson ^ 
(1978) replied to Eggintctn, noting *'yigoroLjiS reactions from people who con- 
sidered Eggintoh* guilty of poor scholarship and c^elesjness with the facts,,, 
^^puttlng his finger on the^ scale .while weighing the evidence** (p, 87), 
Swanson considered the report so lacking in credibility that it could be dis- 
"'missed without a* response; *^It is hardly necessary to reweigh it her#i it 



probably should have been lopked at: more critically, before being put oh: the 
soale in the. first place" (p.^S?). ' ^ ' 
i ... . . , " ■■ ■ ^ ' . ■■ ■ -^^ ' , ■ . ■ ' L ' 

Ond way of looking at these Examples is 'to' assume that* any unfavorable data 
would be considertd, by the vocational educatioh community as lacking qredibil' 
Ity* ypcfltional educators, like any other established groups are'^^uick to 
recent attack and are vigorous in defending thei^ efforts and a^chievemehtSi 
but they also recognize their om weaknesses when* presented with supportable 
data. Uvaluatlve data which do not/ meet, tbat test are rejected, *. , ^ 



Utility ^ ' / . / ' ' ^ ' ^ 

Another condition for the use of evaluative data* is utility. Availability , 

Hand'utility are closely related. In order to be used, data must be readily • 

available* If the language is too technical, however, they will still not be ^ 

u$ed* When Stevenson and Ward (1973) made a study of information available 

to state leaders in conducting evaluations, they found much of* it unusable.* 

• ' i- * 

Their description of the situation makes an excellent point: 

^ Information theorists refer to error in communications is ^oise. * 
The overall literature o^ evaluation is noisyj has wide gaps, is 
' ■' _ . difficult to follow, is voluminous and contradictory , and ^an 

Tuave a render in a state^of^ustration* , It is easy to under^ 
stand how phrases/ such as --raped by rhetorib" and ''information 
pverloading'* could be applied to this literature. (p, 2) 

Educational researchers have the responsibility to explain what research data, , 
including evaluative data, Knean to nonresjearehers. Research ^ta intended for 
'i^ general audience have little utility unless pres^^ted in plfear, unmis.takable 
and easily understood,^anguage. , Some reports of jhe National AdTvllory Council 
on Vocational Education are examples of clear presentations intended for non- 
* researchers. . ,* 

The importance of utility in evaluative data was the'subject of one of the 
^ sessions in the recent national conference on measuring vocational education 
outcomes* DreWes /n978) statedjhat , "Use and ultimate utility sf a standard- 
lEud vocational eo^ucation data system will depend on whether this system is 
primarily accounting or decision^prierfted" (jf. 5). It was generally agreed 
that most existing state reporting systems arfe accounting oriented^ as is the 
^current U.S, Office o€ pducation Reporting sys,tem upon which most state systems 
# are based, As Drewes points. out, *^The most critical consideration governing 
the use of standardised data at the federAl level will be whether -data are 
UHCil primarily to ensure compliance or "to support^ improved teadership''^ (p. 7). 
Thv iirrference Is In the way they 'are' cpUectdd * analyzed, and presentofl; 
Acw'tiunt i ng-or i ented d|ita^ reveal only a'ggregate numbers, such as enrollments, 
t'HjH'iul i t urpH , plnUcmpnts; iind other measurable aspects oT educatldn. l)cci,i|jon= 
iUMtMitcd data reveal which student>^ anil programs , in which locations, and 
u[uier wha\ c i resist aiWes perform hetteiv than others* 'rhesc data have more 
nt i I i t y i o v adift i n i s t rat i ve due i s io^^s . , i . 
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Qn^ fii^l condltioh,*goveridng;th# usW df^ 
* TKis is probably the nost l^ortdi^aiid the most freqM9nMy^ign0rm^ - 
characteristic of data. Consist^nip fflelinH tjiat data must fie cojppcftlhli^ with 
all other, governing factors; such as itgal requiremehtSj^egtablishW pol Icy , 
ralevant advisory topnittoo rocoraiendations, and, even With Wra views of key 
individuals, Thip does pot mtan that tvaluativb .dkta iftmt be distorted or. 
biased or edited to* agree with eitablished policy. It means that evaluative 
data^must be^^collecteds andlyzed, inti^rpreted, and made- available within the 
'boundaries of pbfe sib le action^ - V* ^ ^ 

. A difficult situitidn 1ft' this reipect has been ereated^^ the .state planning 
^ requirements of the 1976 federal legislatignt Each sttite an must r 

; -. , , _ ■ ' . , ' [ ■ . / . . . ' / - " 

Set out explicitly the goal fe the State vf ill seek to achieve*.. in 
- terms of the courses and pthe^, tTaining opportunities to be offered, 
the allocations of-respons ibi 11 tyj for the 'offering of those 
Courses and training opportunities ttmong the vartbus lev^is of 
education and among the various Inst ittitlons of the State , and the 
allo.catlpns of all local, St^te, anrf.Federal financial resources 
^ ^ available in the State amoi^ these courses and •training oppdrtu=^ 
nities, levels of education! and ihstitutions within the Static, 
(P.L. 94-482* SeCl22) ^ \ ^ ^ — ^ . ^ 

It would be easy t^ assume that the flow of^ evaluative data to the'^tpte ; 
agencyj in this kiiid ^f detail vouldfetiable the planneri to^ake single \ 
decisions on program offerings and financial resources t However^ local ' • - 
school districts, not states, usually decide what courses to offer and how 
to distribute ^heir resources* As Drewes* (1978) poirtts out , "Although-^the 
States have constitutional authority for educatibn, they have in effect dele- 
gated ^'their authority to sthe local levelV (p. 8), Evaluative data, in order 
to bp ^consistent with the legil and practical aspects of planning, must 
^ be 'directed ^ward local schools as well as state and^^lderal offlc*e$, /;This , 
"is pspecially true Hince^the resources which support the systgrn are contfo^lled 
more 'often ^t the local 1 eve L than at the state level, ^rfewes states ^that, 
"because of state law, r State' Board of Education Tpoiicies; and the relationship 
of vocational educatitin to gener&l ed^ucation, there is relatively little dis^ 
erotionary authority ov^r the floK of Federal and -dtate funds to the locals** 
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.USES OF ^EVXIUATIVE DATA BY \/OCATTONAL 'EDUCATORS 

The pbtentlal u?c' of , data. gatheVed'^ should be a corisidetatioa in the ^manner in 
which thfey are obtained and the ^alyses and interpretations made. There is. 
no feedback ^ on whethM some evaluative data are^more useful than otheTS. The 
production of evaluative data becomes an ongoing linear activity without 
benefit of its own product evaluation. * * 

Vocational educators use evaluative data in determining policy, establishing 
go&tS\and objective^, developing plans and planning detail, taking adminiS' 
trative action^ allbcatlng and re^al locating funds, obtaining addit'ipnal 
funds, adding or dropping courses and programs, changing curricula, changing 
enrolimont and completion requirements, and public relations. All except the 
last. activity are recognized as legitimate actions supported by evaluative 
data. 



Public Relations 

Public relations, however, can also be an importan^t and entirely proper use of 
evaluative data/ If such data show, for example, that certain programs are 
performing well, and their ^continuation depends on public support , /what better 
use of the data dould be made than to^ help obtain that supportt On the con- 
trary, if a program is politically entrenched and not performing well, evalu- 
ative 'data can be used to inform the public in order to change the situation. 
Admittedly, most public relations based on evaluative data can 'be expected to 
support existing programs rather than to undermine them/ 

Researchers and eva^iMition specialists need to understand the value voca- 
tional educators place on evaluative data for public relations purposes so 
that data ean be' used for this purpose without being easily misrepresented. 
Neither the Wilms nor the Egginto^ Jstu# cited earlier may have been intended 
to destroy public confidenco In voca&onal education, but in each case, the 
pr.esentation of the data made the studies vulnerable to misrepresentation. 

^Somfc of the best examples of the uses of evaluative data, especially for 
public relations, are reported in Volume V of the Project Baseline series 
(Lee. 1976). When recipients of the preyious year's report (Lee, iSJJ] were 
oMod how the data were used, some of the responses were as follows: 

'-Much of the data that the National Advisory Council on Vocational 
Education ha^ used In publications and testimony could be traced 
directly to*your reports ... .Quite frequently this information has , 
been 'sent to pcpplo who call the American. Vocation Association 
Office for information about vocational education programs.... 
Work done by Project Baseline has brought us a long way down the 
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road to gaining national, perspective of vocational education while 

giving the states some basis for comparison, .We (state agencies) 

have used the reports in drawing up goals antf. missions and for the 

friviiion when preparing budget' requests, (pp* 94-107)' 
". ' . • .J 

State Planning . Y , . 

Some ©f the uses of the Baseline data noted were actions, not just knowledge, 
these actions included ptiblic relationSi legislative testimony, and proposal 
writing for project funding. In most cases the action is Implied, not stated. 
The most frequently stated action of the Baseline data was planning and pre^ 
paring state plans. The actual Uie of evaluative data in state, planning 
presumably would be in the allocation of 'resources and" concent rat ion of effort 
on the basis of what the data show, as for example, existing strengths and 
weaknesses, gaps and opportunities, and supply and demahd. O'Reilly (197S) 
found that that is thtf way state agency personhel ttfrtK to describe the plan- 
ning process: ^ , 

The general procedure followed in writing the State plan consists 
of three major steps. The statistical d§ta necessary for evalu- 
ating accomplishments, identifying needl, and establishing goals 
and objectives, are gathered. T^e goals, objectives, and accom- 
plishments of the previous year are reviewed. The goals and 
objectiiX^s for the coming year are established, and a State, plan 
is written. The exact manner and sequence in which these steps 
are completed varies from State to Strfte, and many States have 
elaborated on the basic procedure, (p, 11) 

Only those who. have the responsibility for planning in each of the states can 
verify the process. Little has been written on the subject of the use of 
evaluative data* According to Lawrence and Dane (1974), "An examination of 
the literature reveals that while there are innumerable articles describing 
the development of planning systems, the pilot testing of planning syslfems, 
and the availability of planning systems," there is a serious shortage of 
information on how planning is actually working in ,sltu" (p. 4), They comment 
that ''planning In operation may be so detailed that it is not easily described* 
Suttlor (in O'Roillys 1975) comments further that: 

■ Planning is not a one-shot deal which occurs in a vacuum, performed 
by individuals divorced from where the action is,.., The process 
is Qompletely immersed into all operational aspects of the occu- 
\ pational education program at all levels--f(ideral, stiite, and 
^ local.,,. In the field of occupational education, which operatfs 
within the educational structure with its parameters determined 
by forces, agencies and institutions outside of the educational 
tfv:^ structure, you have an extremely complex and constantly changing 
'tS^ - arena, Cp. 5) • 
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EXTENT TO WHICH VOCATlOhfAL EDUCATORS USE EVALUATIVE DATA ' 

The (lovernment Accounting Office (GAO) document, Report- to the Congress ^ 
W hat the Role of Federal Assistance for Vocational Educatio n? (1974) * 
implied that states were failing to carry out their planning requirements in 
good faith, It may have been less a matter of good faith than legal and 
practical limitations on what State agencies are able to accomplish, Drewes 
(1978) stages tha^t ''relatively few states are in a position, nor would they 
wish to Impose the state will on the locals'' (p. ^8). 

There is even less available knowledge about how and to what extant evalu- 
ative data are used in local vocational education planning, AgWl1/> .3&rewes 
(1978) doubts that much real planning takes place on the basis of e^^luatlve 
data: - 

DccisionH at the local level are frequently constrained by* scarce 
resources. Since state monies tend to flow according to enroll- 
ment statistics, admin istrktors are reluctant to drop courses ^ 
that are popular and hence paying their way Educational resources 
are frequently not easily transformed ^ into other uses. Staffp 
cquipuiont, uiul facilitlos, once ac(|u i red to support particular 
program offerings, are not easily shifted to accommodate newly 
emerging demands, The scarcity and frequently limited substitute 
ability of resources' tends to restrict the discretion of local 
decision makers and thereby reduce the utility of data to con-^ 
tribute to improved program planning; (pp. 9^10) 

^Sources of Information ^ 

Despite the lack of a systematic bddy of literature on the use of evaluative 
data, .there^ are sources where this kind of information should be available. 
Systematic analyses of state plans over a period of several years will give 
some ind lent ion of the kinds of evaluative data consulted, and more importantly, 
the instancoH when i^rogram changes followed directions indicated by the data. 
Studies of this kind at both state "and local levels can be useful in deter- 
mining /Which investments in evaluation justify their costH. , 

Anpther .source of information on the use of evaluative data is the group of 
planners and administrators in each state who manage programs and resources. 
O'Reilly (1975) asked state planners about the kinds and sources of data used 
in writing the state plan,, He found' thf^t thirty-eight states used their 
vocational education statistical data in the planning process. Only twenty- 
nine were using their State Eraployment Service data. Fourteen were using 
additional data from the U.S. Department of Labor, usually the Bureau of Labor 
Statistics, Only eight professed to using data from their state departments 
of economic apd business development. 



Most of the data listed above are not evaluative data, and no Inference in to 
be made frbm them regarding the extent tp which, evaluative data are being - 
i^s6d by state vocational education planners. They simply* illustrate that 
state administrators and planners are a good source of information on the use 
of evaluative data*^As a source they have limitations , however,. O'Reilly 
(1975) was^ unable to tlnd out the ext-ent to which using any of the data, 
including evaluative data, me|nt.ithat changes were made in programs and the 
allocation of resources, or i'rf^the rosultsi *'It is impossible to determine in 
a comprehensive manner-the effect of State Plannihj , upon programs, expenditure 
and program results" (p. iv^* J ^ 

Local^school administrators should be an*%Yeri\ better source of information 
on the influence uf evaluative data on planning or management than sjtate 
plans and state personnel. They could clarify whether changes are carried 
out as planned and directed at the state level, or whether, as Dr^wes (197B) 
insists, they have more authority than the state agency. * 

Other sources to b^^tised with caution are reports of state and local evalua- 
tive systems such as PRIDE in Ohio, the Minnesota follow-up, the Employment/ 
Enrollment Forecasting Guide in Washington State, , OTIS in Oklahoma, and 
others. The originators and directors of systems like those probably know 
how and to what extent the data they provide are used. They may, howei|er, 
lack documentation to support what they say. In this case questions ^ 
asked of the person who created or operates the system. Pride of authorship 
or ownership is powerful incentive for these individuals to err on the side 
of overstatements Such project directors represent a source worth exploring* 
but not without collaborative evidence, . 

Although it may be di'fficuit to assess the use of evaluative data by voca- 
tional educatorSi these data do seem to be used by Congress and many state 
legislatures, A former staff member of the Senate Education Subcommittee 
told the Baseline staff, /4n perusing the reports, I came across statistics 
which indicated that the states were not using vocational education funds in 
accord with the law. Together with the information thrtt jiy col leagues in the 
House had gathered, this resulted in the GAO study and report on vocatianal 
education which later evolved into legislative action-' (Le^, Hl7r?, , 



Inferences ' ' 

Some knowledge of the use of evaluative data by vocational educators could be 
obtained, by inference from two developments for which data are available. 
Changes are taking place in many of the. states, and the nature of the changes 
fitay be related* to evaluative data. In eight states, for example, the number 
of vocati'onal students who are disadvantaged increased between 1971 and 1976 
by 25, 5, 18.7, 9.7, 8.1, 8.1, 7,9, 7.8, and 7,7 percentagb points while 
nationally the percentage declined almost one percentage point (Status of ^ 
Vocatianal Bducration, gchool Year j975-197 6, p, 176). Seven states more than 
doubled Their vocational education enrollments per 1,000 population during the 



same period compared to. a national increase of* S9 percent (p^,< 1563 * Evalu- 
ative data which suggested the need to achieve such ihcreases Afere available ^ 
in those states each year', "Whether of npt there was a direct cause and . 
effect relationship, is only specuration, bat could be determined through " . /^ 
further inquiry, \ ^ 

In various states, substantial increases can be identified in the numbers, of 
handicapped persons served, by vocational education, the number of femal'ei in 
male-dominated programs, the percentages of postsecondary and adult 
enrollments in the annual totals, and shifts in enrollment percentages from 
one occupational area to another. Ill each case, evaluativ^#data were 
available calling for actions which could have produced tTOse rjisults. The 
evaluative data may have b^en the principal cause of such actions, or they 
may have been* only incidental, or they may not have been known by those re- 
sponsible for the actions. There is also the possibility fhat no overt, ^ ^ 
actions were taken and that the changes occurred simply as a result of 
circumstances over which no one exercised appreciable control. 

' ■ ' ■ 

dbviuusly, inferences are a risky way to assess the ext^t to which evaluative 
Until nri^ usud by Viicutional educators. ,Thc second iluvelUpment^ fur which data 
are availiible and from which inferences may^ be made is ev^en less satisfactory 
than indivlduar state performances,, On the basis of the /same performance 
data, there was virtually no improv.ement between 1971 and^ 1976 in the per = 
cent of handicapped or ^disadvantaged students enrolled, ahd very little in 
nontraditional enrollments. Little change occurred in overall enrollment 
and expenditure patterns in vocational education fStatus of Vocati onal 
E ducation, Schodl Year 1975-1976, pp. 158-207), The inference is that 
evaruative data were beiTig used very little, although this feonclusion con- 
flicts with the inference from individual state performances. 

The answer may be tHat in different states evaluative data are b^ng used, 
but in selective ways, so that some states show progress in one direction 
and others in different directions. Tt is more likeiy., however, that 
evaluative data are n|ot being used at nil in some states, and only partially 
in others. In any c^i^se, changes that are evident from individual state data 
tend to balance out nationally, which means that improved performance in one 
state hris been offset by regression in another. Vocational educators may 
have evaluative data which they are unable to use because they lack the j • 
resources to do so. This is what needs to be known. If evaluative data 
have not been used, researchers should inves'tigate the. reasons for this. 



EFFECTIVENESS IN THE USE OF EVALUATIVE DATA 

Only whore the extent of use of evaluative data is known can the effectiveness 
of these data be examined, This Is the ultimate tost of the worth of 
evaluation.. Unfortunately, it is a stop that depends on finding out how and 
to what extent vocational educators arc us-ing evaluative data. 
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Th© sources of this knowledge are 'local , statej and national evaluative- data. 
The lyrocess^ is cyclicalj bfiginning with the collection of .evaluative data 
about program perfotmange* Lawifence and Dane (.1974) developed a prototype 
of voca^^onal' education plailning ^and suggested the foncrwingi * . 

There is "a cyclical ^process operating in which the adtions of 
X plan^iers influence those circumstances toward which planning is , ^ " 
directi^, and the effects of planning/are, continually being 
reintroduced to the . planner as environmental changes Jf or him to 
take into account in his next -planning cycl;B* The crux of this 
theoretical position rests in the view of the planning process 
as dynamic and ijfcitinuali and it implies the h^^^ for a regular - 
flow of irifcrmation. between the 'planning sj^tem ind its associated 
environment* (p* 8) ^ , 

Lawrence and Dane used a national conference to develop their planning models 
This mighit tonstitute an appropriate means of developing a research model ^ 
for assessing the use and effcctivenfess of evaluati^ data. The present^ 
report is a review of knowledge about current practices suggest infe omi s's lon^ 
In the research* The work to be done wduld benefit from the Iftput of evalua- 
tion t^pccialistSj admi n Istrators s planners, and researchers. In the meantime^ 
vocational education evaluato^'s and producers of evaluative data may want to 
make initial inquiries into the best means of assessing the effectiveness of 
the use of evaluative data. ' ^ 

< . • a ■ " ' 

OBSERVATIONS ABOUT CURRENT "EVALUATIVE DATA ^ , 

*The most extensive evaluative data for use by , vocational educators are 
produced by state and locar management information systems (MISs) , These 
usually include what the states *are'^ required to collect for the U.S. Office 
of Education, They do -have some value, even though t^ey are compliance 
oriented and have limitations affecting their reliability and utilization. 
They do provide a fairly^' reliable count of students enrolled by occupational 
programj which can be examined each year. They contain"" somewhat less rcliablej 
but nevertheless useful s placement data* They include aggregate breakouts by 
sexj handicapped, disadvantagedj and dthnlc minorities^ as well as expendi* 
ture data which arc quite accurate as far as federal and state support Is coiV- 
corned, ^ ^ 1 ' . 

A few of the states have taken= a ftep toward btetter evaluative data in their 
MISs by collecting unit level data. This meanK enrollment data by individual 
student characteristics,' such, as age, ^ex, handi^pped or disadvantaged, • ^; 
ethnic minorityi occupational programj grade levels institution, and name and 
address.' Systems with unit data are always autoiHaMd, since the quantities ' 
of data to be sorted and retrifcvod are too great for^manual handlingi A 
number of states >have unit data follow-up systems, some of which can match 
enrollment with follow-up results. The sophistication of the automated systems 

. . ■ ■ ■ ' ^ c~ ' 
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!tmt advanced rapidly, with the result that tHe potential for their use is 
grWter than their actual utilizati :^ ■ - 

Utilization gf th^ data ^techip logy has been encouraged 'through vocational 
education research and; devfelgpment funds ^at ^th the state and federal 'le;vela. 
The field as, a . whole has some distance to before highly useful eyaluative 
data can be expected from the state syst^s,^ Starr (1975) comments that: 

A review of the literature on th^develppments in management infor- 
niation- systems for vocajtional education indicate that, to V consid- 
erable extent, many SDVE's continue to bpera^ with second and 
third generation systems* In syc^ systems a disproportionate decree 
of emphasis^ continues to be .placedi on the que &t for the impospible 
dreamr that of identifying, the aliusive,^ simple^ common core of 
unchanging daja elements which cari'^be supplied with minimuni effort 
by the local school sources, and which will provide ujers with the 
maximum data needed for decision making with regaTd to policy 
formulation^ operations and control for accountability purposes , 
What is required is a fourth generation system* Cpp- 2-3) , 

^Starr, Bladk, and Gray (1977) have stated that "anly five states were found 
to have MISs that had a combination of very good dat'a base, data processings^ 
and report generating capabilities" (p, Ij, In an overall assessmentj they 
found: ^ . ^ # . 

It is apparent that the s.tato MISs studied were neither compre-^ 
hensive ncp adequate within the framGWork of the standards- set 
by the project. The only ^eal strength was in ha'rdware support. 
The use of at least a meditim-size computer in 81% of the states 
. indicates that availabirity of computer facilities is not a 
serious problem though poor service and lack of ready access was 
often founJ to be a constraint in their use. Even in states whi^ch 
. . do not use com'ppters, there were usually computers available which 
could be flSed. *The operut Ing components were, oh the average, 
quite Inadequate with only 28% of states having at least near- 
adequate data output, 19% having ' adequate data processi ng and 11% 
having at least nenr-adequato datirhasoH* Support .components 
ware almost non-ox istejit / No sjgte h^ad even near-adequnte docu- . 
mentation or assessment, and onflrl'o'had at least near- adequate 
. training. 29) ^ • 

The major reason for the existence on state MISs i^ to pMduce statistical 
reports each ye^r for the federal government, which in turn are the input for 
an annual statistical . document prepared by tjie U.S, Office of Education, As 
a source of evaluative data, this document, serves an elementary purpose, but 
has never been widely used for program planning, gjariagement , or even legis- 
lative action. It is compliance oriented and suffers from lack of interpre- 
tation while containing uneven and nojistandard data, A parallel set of 
documents containing evaluative data are state descriptive reports, and these 
are often useful for detail. They ^o not serve. 'as objective indicators of 
program accohiplishment , but as illustrative material for the statistical 



report?. In 1979', under the 1976 legislation^ aiinuat sjate evalMation reports' 
will be .reqijir©y^whi^h are expfected to be dmportant sources of evaluative J 
datA. Their reUabiMty, credibility / and utiH still unknown, ahd 

they will be rj^gorously, scrutinized, v » " 

' ■ . . ' • ' ■■ ' ' ' •■ ^ . ' 

Other soujqes of existing eval'uative data 'fo^ Vocational educators arc the ^ 
reports of State Advisory Counci Is; for Vocational Elducatlcfn (SACVlis) nnd the 
Natidnal Advisory Council for TVocational Education (NACVE), projoct BnsoHno 
was i product of NACVE, since the Congressional mandate under which it was • 
carried out gav0 the fespbnsibility^to the. National Councl 1 : The SACVP. ■ ' 
reports, and an annual NACVE summary of t^ose r^pQrt's , are of mixed quality » 
For the most part the SACVEs have not conducted evaluation studies, relying 
instead on observation and the examination of available evaluative data, 
including state descriptive and statistical reports* Under the 1976 legisla- . 
tion their evfjuati^ responsibilities are more specific, but their sources 
of data are still largely limited ttf what tfiey can obtain from the agencies 
whose performance they are evaluating^ ^ ■ 

Tills may change with^the implementatioB-bf a new national vocational educa- 
tion data system (VEDS) , also mandated fay the 1976 legislation under the 
responsibiHty of the^ National Center for Educatldnal Statistics (NCES) * The ' 
promise of VEDS is unlimited, i^s fulfillment dependent on the willingness of*' 
Cortgress anc the states to provide fund^ for development and ^n the willing- 
ness of agencies and^individuals to establish a major neviv cooperative enter- 
prise* Slow but definite progress hri been made, and the prognosis is 
guardedly hopeful * The most seripus problem ^faced by boih the states and 
NCES is inadequate funding from Cdngress, ^ _ 

Finally, a potential source of evaluative data ^anSSanother product pf the j 
1976 le^slation is t'hd National Occupational' Information Coordinating 
Commi^fe (NOICC) and a similar committee (SOICC) in\ach of the states. As 
yet, nOICC and the SOICCs have no clear legislative direction in .reaching * 
their objectives. They are slowly getting organised, and while ideally they 
could brings about the interchange of data among several federal Ij^ supported 
training systems, (vocational education, CETA, vocational rehabilitation, and 
apprenticaship) , there is no indication yet that they , will do much more thajri" 
tp^observe and advise. . 
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A number of co^itiohs must be met before evaluative data can be used. The 
fijst of these rs availability, which means the data must reach those who are 
in positions to use them* The second is^ reliability . The third condition is 
credibility, often related to buf-not the same as reliability. Data may be 
reliable, but'\lf they are n^ trusted they will hot ^be usedi Utility is a 
fourth condition, and one that pesearchers *and information specialists often 
neglect* If reliable, credible data are made available in a form not clearly 
grasped or under stbod by vocational educators, their use is severely . 



restri\:ted.» And v filial ly^ Jh£?ro^i/fi ccrtisd^stQifcy , EvAluativd^ data are not likely 
to hs used if 'they ^'ar^e inconh^i^^*tent>itti legal requiremenlV^ ^^tabllshed v 
policy, adv^lsory cormikltteo recomn|eni3at\ons , and the ^Qnvicti.©ns. of adminis-,^ 
tratcfrs and pdlicy milkers,* ,v ' W - f - 'j^ ' 

l^i'lu rese^fch Oil tfip^use of ^ evaluativd data by ^ocAtionali educators is ^- 

'Timi ted;/ tlhc^re-i# nevertheless envplrical ovidenc^o^ theii^ use* The uses . ^ 

ma s t 6t> f tt^f^Fqii n d a r u In ^ pub 1 iy re I at 1 on h /ac t i v i t i b s , in *TO^P ^ring -idgisla t.i v e ^ 

-and budget t^niuQstSp in pre^Jnng ciass'ropia material ih.,^wpptional teachur ^, 
ed i It 1 o n ctj^ir^Qfii in p repa ring r e ji o i^fi i^md instate an'J^I »c a 1 planning'- I.n 

-planning, howov^, the use of evaluat i ve^d^ta is so vague and uncertain. thcit 
the extent and c?f feet i veness of that use may^fal l considerably short of Current 
expey tat i unsr^ - ' i *^ / ^ * , ^ § / / 

Sources frbrnKwhidh knowledge about tho^ extent and ef fectrve^ess of using J 
evaluative ^data should be avaiiable^vare: fl) thB annual state planSj (2.> the 
planners and administrators of vocational education resources in ea^h* state, 
(.3) local school admiriistrators,' and (4) -the designers and'^perator/ of st^te 
mana^gement inforniation systems* Inferences can also be made atout the use of 
evaluative drfa from state and national . statistics^nd trend- analysis over a 
period of ye§rs* Congress and probabry state legislatures make exteiisive use | 
of epiluative data. A miscell|py of other known uses suggests that^ researchets 
iind evaluation specialistB shpuld cpns'ider the product of their efforts as 
mul 1 1 -rtargeted rather than directed towai\l' a limi'te^ number of users. , ' 

St*att\ manugem|.Mit information systems which include enrollment, placement , ^ 
staffs and expenditure ^data. contain the mostS fext ens ivQ' evaluative data cur= 
rciit l^y available,^* A few state systtMiiH with unit dafa and analytical capability 
are^ goodi but mos^ suffer from the self-imposed limi^tatioiis of collecting only 
tb^ compliance data required for annuaT repOFts to JJSOE^ Even those with 
automated data processing have often failed to keep up witK the rapid develop- 
ment Qf computer technology* The national and state vqcational education 
advisory council reports are another potential sourcf of evaluative data, but 
with the exception of the NACVE's sponsorship of Project Baselines these - 
soupce^ usual ly contain only observations and data ^upplied by the administra- 
tive agencies" Several provisj^ons of the' 1976 federal legislation are intended 
to increase , both the quality and availability of evaluative dataj notably the 
new national Vocation Education Data Systd^^(VEDS) . ^ ' > 

Qverall, rosoarcher^, ovaluators, and adminstfators should- hot wait until VEDS ^ 
'and otlter products of the 1976 l^egisiatlon mat^orlalize, A better national 
datii system, state and federal eva Tuat ions , NOICC and* the SOlCCs, and more 
stringent planning requ i rement s- may result in better wriliitrt^ivedatsr, but more 
interest will have to be shown in determining whether such dat^i are used. The 
initiative remains to be, tak^n^by advisory councils, evaluation specialists, 
and researchers. . ' c 
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cormnencftitlons . . , * ' / , - * * \ 




,^*Fiire. courses }of actioi) are recommendpd: 

■ \ ' ,x^^ \ ^ ' ' . . 

1., The, National TnstPtptu of Fidticntldn (NIH)> the Nnt loniil .Contor for 
Hescnrcli in Vociitiunnl lidiicntlon, the UHOIi. lUirenu of (iccupnnnnn I nhd 
Adult lidiiiiat ion (lUlAliJ , and thp Htat e UeHjeurch Cnordinnt ing llnils (HCIIsj' 
Hhbuld muku theaiHo o^^ oval native data aijH?cognl zed i^^eareh ^at ek<y»'y ' 
I'h Is sliouUU he giyon high pr^dr ity at T^'st until a huily of^sHile 
knowlDdgc i H entahl i'Shcd. ^ Mr r.- * * v 

valuation specialists should extend their professional expertise and . 
xperience to include^*the use of evaluative data ^ circumstances affecting 
* thkt use" and 4ts extent and effectiveness ^ ' / 

I ^' ' \ * : ^ . ; ^ ^ ^ ' ' 

3. Data^^iyitems operators aftd pther pi*oducers of evaluative data should' 
build *into their systems a,,faedbacH mechanism fpr ^he drial purpose of 

I ' docij^pnting the need for trteir products and of evaluating the effect^ 
\^ ' iveneSsi of those product si . , ^ ' 

4. * AdvisorV touncils at the federal, state, and local levels should 
. ' to make inquiries into the uSe of evaluative datk, the extent of uhq, 

\. r and t*he results obtained by,'administrative, agencies. ^ 

M A national conforeftce on the use of evaluative data by vocational 
_ ' , / educators should be* sponsored by NlE, the National Center, BOAE, or 
/ a combinat?.on of these ah4^^ther sponsors to^ explore wayi and means 
of developing a basic knowledge of the use of evaluative data* A 
research model would be a welcome outcome ?of such exploration. 
, * - ■ - 
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